{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "IuCdVcDAba3u"
      },
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pinecone-io/examples/blob/master/learn/generation/prompt-engineering.ipynb) [![Open nbviewer](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/nbviewer-shield.svg)](https://nbviewer.org/github/pinecone-io/examples/blob/master/learn/generation/prompt-engineering.ipynb)\n",
        "\n",
        "# Prompt Engineering\n",
        "\n",
        "In this notebook we'll explore the fundamentals of prompt engineering. We'll start by installing the `openai` library, which we'll be using throughout these examples. However, note that we can use other LLMs here, like those offered by Cohere or open source alternatives available via Hugging Face."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "2Ck8Jo5_ba3w",
        "outputId": "a9e61507-d1a3-4bd9-d425-13f66301abf2"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\n",
            "Usage:   \n",
            "  pip3 <command> [options]\n",
            "\n",
            "no such option: -U\n"
          ]
        }
      ],
      "source": [
        "!pip install -qU openai==0.27.7"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "0G4_uOtqba3x"
      },
      "source": [
        "## Structure of a Prompt\n",
        "\n",
        "A prompt can consist of multiple components:\n",
        "\n",
        "* Instructions\n",
        "* External information or context\n",
        "* User input or query\n",
        "* Output indicator\n",
        "\n",
        "Not all prompts require all of these components, but often a good prompt will use two or more of them. Let's define what they all are more precisely.\n",
        "\n",
        "**Instructions** tell the model what to do, typically how it should use inputs and/or external information to produce the output we want.\n",
        "\n",
        "**External information or context** are additional information that we either manually insert into the prompt, retrieve via a vector database (long-term memory), or pull in through other means (API calls, calculations, etc).\n",
        "\n",
        "**User input or query** is typically a query directly input by the user of the system.\n",
        "\n",
        "**Output indicator** is the *beginning* of the generated text. For a model generating Python code we may put `import ` (as most Python scripts begin with a library `import`), or a chatbot may begin with `Chatbot: ` (assuming we format the chatbot script as lines of interchanging text between `User` and `Chatbot`).\n",
        "\n",
        "Each of these components should usually be placed the order we've described them. We start with instructions, provide context (if needed), then add the user input, and finally end with the output indicator."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "4Y7XX3i9ba3y"
      },
      "outputs": [],
      "source": [
        "prompt = \"\"\"Answer the question based on the context below. If the\n",
        "question cannot be answered using the information provided answer\n",
        "with \"I don't know\".\n",
        "\n",
        "Context: Large Language Models (LLMs) are the latest models used in NLP.\n",
        "Their superior performance over smaller models has made them incredibly\n",
        "useful for developers building NLP enabled applications. These models\n",
        "can be accessed via Hugging Face's `transformers` library, via OpenAI\n",
        "using the `openai` library, and via Cohere using the `cohere` library.\n",
        "\n",
        "Question: Which libraries and model providers offer LLMs?\n",
        "\n",
        "Answer: \"\"\""
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "gwF4OVNzba3y"
      },
      "source": [
        "In this example we have:\n",
        "\n",
        "```\n",
        "Instructions\n",
        "\n",
        "Context\n",
        "\n",
        "Question (user input)\n",
        "\n",
        "Output indicator (\"Answer: \")\n",
        "```\n",
        "\n",
        "Let's try sending this to a GPT-3 model. For this, you will need [an OpenAI API key](https://beta.openai.com/account/api-keys).\n",
        "\n",
        "We initialize a `text-davinci-003` model like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EXDaIAwmba3y",
        "outputId": "81c6fe3a-b84b-4c70-afa6-32a14620e46c"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "<OpenAIObject list at 0x7f21bce77470> JSON: {\n",
              "  \"data\": [\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"whisper-1\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-internal\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"babbage\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"davinci\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"text-davinci-edit-001\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"text-davinci-003\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-internal\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"babbage-code-search-code\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"text-similarity-babbage-001\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"code-davinci-edit-001\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"text-davinci-001\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"ada\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"babbage-code-search-text\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"gpt-4-0314\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"babbage-similarity\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"code-search-babbage-text-001\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"text-curie-001\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"code-search-babbage-code-001\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"text-ada-001\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"text-embedding-ada-002\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-internal\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"text-similarity-ada-001\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"curie-instruct-beta\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"ada-code-search-code\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"ada-similarity\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"gpt-4\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"code-search-ada-text-001\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"text-search-ada-query-001\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"davinci-search-document\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"ada-code-search-text\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"text-search-ada-doc-001\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"davinci-instruct-beta\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"text-similarity-curie-001\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"code-search-ada-code-001\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"ada-search-query\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"text-search-davinci-query-001\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"curie-search-query\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"davinci-search-query\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"babbage-search-document\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"ada-search-document\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"text-search-curie-query-001\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"text-search-babbage-doc-001\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"gpt-3.5-turbo\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"curie-search-document\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"text-search-curie-doc-001\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"babbage-search-query\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"text-babbage-001\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"text-search-davinci-doc-001\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"text-search-babbage-query-001\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"curie-similarity\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"gpt-3.5-turbo-0301\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"curie\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"text-similarity-davinci-001\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"text-davinci-002\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    },\n",
              "    {\n",
              "      \"created\": null,\n",
              "      \"id\": \"davinci-similarity\",\n",
              "      \"object\": \"engine\",\n",
              "      \"owner\": \"openai-dev\",\n",
              "      \"permissions\": null,\n",
              "      \"ready\": true\n",
              "    }\n",
              "  ],\n",
              "  \"object\": \"list\"\n",
              "}"
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import os\n",
        "import openai\n",
        "\n",
        "# get API key from top-right dropdown on OpenAI website\n",
        "openai.api_key = os.getenv(\"OPENAI_API_KEY\") or \"OPENAI_API_KEY\"\n",
        "\n",
        "openai.Engine.list()  # check we have authenticated"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "trhbZNwWba3z"
      },
      "source": [
        "And make a generation from our prompt."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "NFlrjyFfba3z",
        "outputId": "4f44e343-75de-4303-9ecd-d578945220cf"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Hugging Face's `transformers` library, OpenAI using the `openai` library, and Cohere using the `cohere` library.\n"
          ]
        }
      ],
      "source": [
        "# now query text-davinci-003\n",
        "res = openai.Completion.create(\n",
        "    engine='text-davinci-003',\n",
        "    prompt=prompt,\n",
        "    max_tokens=256\n",
        ")\n",
        "\n",
        "print(res['choices'][0]['text'].strip())"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "C2zcqYnwba3z"
      },
      "source": [
        "Alternatively, if we do have the correct information withing the `context`, the model should reply with `\"I don't know\"`, let's try."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "w0_jElSnba30",
        "outputId": "3837038c-5288-4551-8605-053b61bcf0df"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "I don't know.\n"
          ]
        }
      ],
      "source": [
        "prompt = \"\"\"Answer the question based on the context below. If the\n",
        "question cannot be answered using the information provided answer\n",
        "with \"I don't know\".\n",
        "\n",
        "Context: Libraries are places full of books.\n",
        "\n",
        "Question: Which libraries and model providers offer LLMs?\n",
        "\n",
        "Answer: \"\"\"\n",
        "\n",
        "res = openai.Completion.create(\n",
        "    engine='text-davinci-003',\n",
        "    prompt=prompt,\n",
        "    max_tokens=256\n",
        ")\n",
        "\n",
        "print(res['choices'][0]['text'].strip())"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "G8RjQ3OSba30"
      },
      "source": [
        "Perfect, our instructions are being understood by the model. In most real use-cases we won't be providing the external information / context to the model manually. Instead, it will be an automatic process using something like [long-term memory](https://www.pinecone.io/learn/openai-gen-qa/) to retrieve relevant information from an external source.\n",
        "\n",
        "For now, that's beyond the scope of what we're exploring here, you can find more on that in the link above.\n",
        "\n",
        "In summary, a prompt often consists of those four components: instructions, context(s), user input, and the output indicator. Now we'll take a look at creative vs. stricter generation.\n",
        "\n",
        "## Generation Temperature\n",
        "\n",
        "The `temperature` parameter used in generation models tells us how \"random\" the model can be. It represents the probability of a model to choose a word which is *not* the first choice of the model.\n",
        "\n",
        "This works because the model is actually assigning a probability prediction across all tokens within it's vocabulary with each _\"step\"_ of the model (each new word or sub-word).\n",
        "\n",
        "With each new step forwards the model considers the previous tokens fed into the model, creates an embedding by encoding the information from these tokens over many model encoder layers, then passes this encoding to a decoder. The decoder then predicts the probability of each token that the model knows (ie is within the model *vocabulary*) based on the information encoded within the embedding.\n",
        "\n",
        "At a temperature of `0.0` the decoder will always select the top predicted token. At a temperature of `1.0` the model will always select a word that *is predicted* considering it's assigned probability.\n",
        "\n",
        "Considering all of this, if we have a conservative, fact based Q&A like in the previous example, it makes sense to set a lower `temperature`. However, if we're wanting to produce some creative writing or chatbot conversations, we might want to experiment and increase `temperature`. Let's try it."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ACaf04zjba30",
        "outputId": "2ccb4263-8fd2-44f5-ae0c-73a20490ef78"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Oh, just hanging out and having a good time. What about you?\n"
          ]
        }
      ],
      "source": [
        "prompt = \"\"\"The below is a conversation with a funny chatbot. The\n",
        "chatbot's responses are amusing and entertaining.\n",
        "\n",
        "Chatbot: Hi there! I'm a chatbot.\n",
        "User: Hi, what are you doing today?\n",
        "Chatbot: \"\"\"\n",
        "\n",
        "res = openai.Completion.create(\n",
        "    engine='text-davinci-003',\n",
        "    prompt=prompt,\n",
        "    max_tokens=256,\n",
        "    temperature=0.0  # set the temperature, default is 1\n",
        ")\n",
        "\n",
        "print(res['choices'][0]['text'].strip())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ZVacrQseba30",
        "outputId": "ed2f9c9d-5f66-4f4b-dbd2-77e01c679e65"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "I'm making people smile! What about you?\n"
          ]
        }
      ],
      "source": [
        "prompt = \"\"\"The below is a conversation with a funny chatbot. The\n",
        "chatbot's responses are amusing and entertaining.\n",
        "\n",
        "Chatbot: Hi there! I'm a chatbot.\n",
        "User: Hi, what are you doing today?\n",
        "Chatbot: \"\"\"\n",
        "\n",
        "res = openai.Completion.create(\n",
        "    engine='text-davinci-003',\n",
        "    prompt=prompt,\n",
        "    max_tokens=512,\n",
        "    temperature=1.0\n",
        ")\n",
        "\n",
        "print(res['choices'][0]['text'].strip())"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "WJzslropba31"
      },
      "source": [
        "The second response is far more creative and demonstrates the type of difference we can expect between low `temperature` and high `temperature` generations.\n",
        "\n",
        "## Few-shot Training\n",
        "\n",
        "Sometimes we might find that a model doesn't seem to get what we'd like it to do. We can see this in the following example:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "PHUeF3KHba31",
        "outputId": "ac694101-a350-42e0-c7fe-c65dabf2e67b"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "The meaning of life isn't something that can be summed up in a few sentences. It's something that each person must discover for themselves.\n"
          ]
        }
      ],
      "source": [
        "prompt = \"\"\"The following is a conversation with an AI assistant.\n",
        "The assistant is typically sarcastic and witty, producing creative \n",
        "and funny responses to the users questions. \n",
        "\n",
        "User: What is the meaning of life?\n",
        "AI: \"\"\"\n",
        "\n",
        "res = openai.Completion.create(\n",
        "    engine='text-davinci-003',\n",
        "    prompt=prompt,\n",
        "    max_tokens=256,\n",
        "    temperature=1.0\n",
        ")\n",
        "\n",
        "print(res['choices'][0]['text'].strip())"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "M50njhtWba31"
      },
      "source": [
        "In this case we're asking for something amusing, a joke in return of our serious question. But we get a serious response even with the `temperature` set to `1.0`. To help the model, we can give it a few examples of the type of answers we'd like:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "O2IPuMnLba31",
        "outputId": "59a96dea-de49-4040-c2b7-5e925a4354b1"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "All I can say is 42...just kidding! The meaning of life is the journey to find your own.\n"
          ]
        }
      ],
      "source": [
        "prompt = \"\"\"The following are exerpts from conversations with an AI assistant.\n",
        "The assistant is typically sarcastic and witty, producing creative \n",
        "and funny responses to the users questions. Here are some examples: \n",
        "\n",
        "User: How are you?\n",
        "AI: I can't complain but sometimes I still do.\n",
        "\n",
        "User: What time is it?\n",
        "AI: It's time to get a watch.\n",
        "\n",
        "User: What is the meaning of life?\n",
        "AI: \"\"\"\n",
        "\n",
        "res = openai.Completion.create(\n",
        "    engine='text-davinci-003',\n",
        "    prompt=prompt,\n",
        "    max_tokens=256,\n",
        "    temperature=1.0\n",
        ")\n",
        "\n",
        "print(res['choices'][0]['text'].strip())"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "2HsKGgknba31"
      },
      "source": [
        "This is a much better response and the way we did this was by providing a *few* examples that included the example inputs and outputs that we'd expect. We refer to this as _\"few-shot learning\"_.\n",
        "\n",
        "## Adding Multiple Contexts\n",
        "\n",
        "In some use-cases like question-answering we can use an external source of information to improve the reliability or *factfulness* of model responses. We refer to this information as _\"source knowledge\"_, which is any knowledge fed into the model via the input prompt.\n",
        "\n",
        "We'll create a list of \"dummy\" external information. In reality we'd likely use [long-term memory](https://www.pinecone.io/learn/openai-gen-qa/) or some form of information grabbing APIs."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "id": "QhiAZbxMba31"
      },
      "outputs": [],
      "source": [
        "contexts = [\n",
        "    (\n",
        "        \"Large Language Models (LLMs) are the latest models used in NLP. \" +\n",
        "        \"Their superior performance over smaller models has made them incredibly \" +\n",
        "        \"useful for developers building NLP enabled applications. These models \" +\n",
        "        \"can be accessed via Hugging Face's `transformers` library, via OpenAI \" +\n",
        "        \"using the `openai` library, and via Cohere using the `cohere` library.\"\n",
        "    ),\n",
        "    (\n",
        "        \"To use OpenAI's GPT-3 model for completion (generation) tasks, you \" +\n",
        "        \"first need to get an API key from \" +\n",
        "        \"'https://beta.openai.com/account/api-keys'.\"\n",
        "    ),\n",
        "    (\n",
        "        \"OpenAI's API is accessible via Python using the `openai` library. \" +\n",
        "        \"After installing the library with pip you can use it as follows: \\n\" +\n",
        "        \"```import openai\\nopenai.api_key = 'YOUR_API_KEY'\\nprompt = \\n\" +\n",
        "        \"'<YOUR PROMPT>'\\nres = openai.Completion.create(engine='text-davinci\" +\n",
        "        \"-003', prompt=prompt, max_tokens=100)\\nprint(res)\"\n",
        "    ),\n",
        "    (\n",
        "        \"The OpenAI endpoint is available for completion tasks via the \" +\n",
        "        \"LangChain library. To use it, first install the library with \" +\n",
        "        \"`pip install langchain openai`. Then, import the library and \" +\n",
        "        \"initialize the model as follows: \\n\" +\n",
        "        \"```from langchain.llms import OpenAI\\nopenai = OpenAI(\" +\n",
        "        \"model_name='text-davinci-003', openai_api_key='YOUR_API_KEY')\\n\" +\n",
        "        \"prompt = 'YOUR_PROMPT'\\nprint(openai(prompt))```\"\n",
        "    )\n",
        "]"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "8pJ82t7Lba31"
      },
      "source": [
        "We would feed this external information into our prompt between the initial *instructions* and the *user input*. For OpenAI models it's recommended to separate the contexts from the rest of the prompt using `###` or `\"\"\"`, and each independent context can be separated with a few newlines and `##`, like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "xnUQvuIiba32",
        "outputId": "92fccb2a-d533-4ddd-dee5-1a2c4ee305a4"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Answer the question based on the contexts below. If the\n",
            "question cannot be answered using the information provided answer\n",
            "with \"I don't know\".\n",
            "\n",
            "###\n",
            "\n",
            "Contexts:\n",
            "Large Language Models (LLMs) are the latest models used in NLP. Their superior performance over smaller models has made them incredibly useful for developers building NLP enabled applications. These models can be accessed via Hugging Face's `transformers` library, via OpenAI using the `openai` library, and via Cohere using the `cohere` library.\n",
            "\n",
            "##\n",
            "\n",
            "To use OpenAI's GPT-3 model for completion (generation) tasks, you first need to get an API key from 'https://beta.openai.com/account/api-keys'.\n",
            "\n",
            "##\n",
            "\n",
            "OpenAI's API is accessible via Python using the `openai` library. After installing the library with pip you can use it as follows: \n",
            "```import openai\n",
            "openai.api_key = 'YOUR_API_KEY'\n",
            "prompt = \n",
            "'<YOUR PROMPT>'\n",
            "res = openai.Completion.create(engine='text-davinci-003', prompt=prompt, max_tokens=100)\n",
            "print(res)\n",
            "\n",
            "##\n",
            "\n",
            "The OpenAI endpoint is available for completion tasks via the LangChain library. To use it, first install the library with `pip install langchain openai`. Then, import the library and initialize the model as follows: \n",
            "```from langchain.llms import OpenAI\n",
            "openai = OpenAI(model_name='text-davinci-003', openai_api_key='YOUR_API_KEY')\n",
            "prompt = 'YOUR_PROMPT'\n",
            "print(openai(prompt))```\n",
            "\n",
            "###\n",
            "\n",
            "Question: Give me two examples of how to use OpenAI's GPT-3 model\n",
            "using Python from start to finish\n",
            "\n",
            "Answer: \n"
          ]
        }
      ],
      "source": [
        "context_str = '\\n\\n##\\n\\n'.join(contexts)\n",
        "\n",
        "print(f\"\"\"Answer the question based on the contexts below. If the\n",
        "question cannot be answered using the information provided answer\n",
        "with \"I don't know\".\n",
        "\n",
        "###\n",
        "\n",
        "Contexts:\n",
        "{context_str}\n",
        "\n",
        "###\n",
        "\n",
        "Question: Give me two examples of how to use OpenAI's GPT-3 model\n",
        "using Python from start to finish\n",
        "\n",
        "Answer: \"\"\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_UMY42yAba32",
        "outputId": "dec3027f-73c3-4f76-b911-96c8ea355996"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "1. import openai\n",
            "openai.api_key = 'YOUR_API_KEY'\n",
            "prompt = '<YOUR PROMPT>'\n",
            "res = openai.Completion.create(engine='text-davinci-003', prompt=prompt, max_tokens=100)\n",
            "print(res)\n",
            "\n",
            "2. from langchain.llms import OpenAI\n",
            "openai = OpenAI(model_name='text-davinci-003', openai_api_key='YOUR_API_KEY')\n",
            "prompt = 'YOUR_PROMPT'\n",
            "print(openai(prompt))\n"
          ]
        }
      ],
      "source": [
        "prompt = f\"\"\"Answer the question based on the contexts below. If the\n",
        "question cannot be answered using the information provided answer\n",
        "with \"I don't know\".\n",
        "\n",
        "###\n",
        "\n",
        "Contexts:\n",
        "{context_str}\n",
        "\n",
        "###\n",
        "\n",
        "Question: Give me two examples of how to use OpenAI's GPT-3 model\n",
        "using Python from start to finish\n",
        "\n",
        "Answer: \"\"\"\n",
        "\n",
        "res = openai.Completion.create(\n",
        "    engine='text-davinci-003',\n",
        "    prompt=prompt,\n",
        "    max_tokens=256,\n",
        "    temperature=0.0\n",
        ")\n",
        "\n",
        "print(res['choices'][0]['text'].strip())"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "ScZ9qE-Lba32"
      },
      "source": [
        "Not bad, but are these contexts actually helping? Maybe the model is able to answer these questions without the additional information (source knowledge) as is able to rely solely on information stored within the model's internal parameters (parametric knowledge). Let's ask again without the external information."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "alevc66Kba32",
        "outputId": "73d67832-f531-407d-afd0-13e99070a334"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "1. Using OpenAI's GPT-3 model with Python to generate text: \n",
            "    - Install the OpenAI Python package\n",
            "    - Load the GPT-3 model\n",
            "    - Generate text using the GPT-3 model\n",
            "\n",
            "2. Using OpenAI's GPT-3 model with Python to generate images: \n",
            "    - Install the OpenAI Python package\n",
            "    - Load the GPT-3 model\n",
            "    - Generate images using the GPT-3 model\n"
          ]
        }
      ],
      "source": [
        "prompt = f\"\"\"Answer the question based on the contexts below. If the\n",
        "question cannot be answered using the information provided answer\n",
        "with \"I don't know\".\n",
        "\n",
        "Question: Give me two examples of how to use OpenAI's GPT-3 model\n",
        "using Python from start to finish\n",
        "\n",
        "Answer: \"\"\"\n",
        "\n",
        "res = openai.Completion.create(\n",
        "    engine='text-davinci-003',\n",
        "    prompt=prompt,\n",
        "    max_tokens=256,\n",
        "    temperature=0.0\n",
        ")\n",
        "\n",
        "print(res['choices'][0]['text'].strip())"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "b_UCtq-qba32"
      },
      "source": [
        "These are not really what we asked for, and are definitely not very specific. So clearly adding some source knowledge to our prompts can result in some much better results.\n",
        "\n",
        "## Maximum Prompt Sizes\n",
        "\n",
        "Considering that we might want to feed in external information to our prompts, they can naturally become quite large. With this we need to ask how large our prompts can be, because there is a maxiumum size.\n",
        "\n",
        "The maxiumum *context window* of a LLM refers to tokens across both the *prompt* and the *completion* text. For `text-davinci-003` this is `4097` tokens.\n",
        "\n",
        "We can set the maximum completion length of our model using `openai.max_tokens = 123`. However, measuring the total number of input tokens is more complex.\n",
        "\n",
        "Because tokens don't map directly to words, we can only measure the number of tokens from text by actually tokenizing the text. GPT models use [OpenAI's TikToken tokenizer](https://github.com/openai/tiktoken). We can install the library via Pip:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 14,
      "metadata": {
        "id": "Z3F1ht1Dba32"
      },
      "outputs": [],
      "source": [
        "!pip install -qU tiktoken==0.4.0"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "xoPgwCenba33"
      },
      "source": [
        "Taking the earlier prompt we can measure the number of tokens like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 15,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yNgL_NPLba33",
        "outputId": "0ba596b8-d1da-4c14-9642-26e544e70428"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "412"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import tiktoken\n",
        "\n",
        "prompt = f\"\"\"Answer the question based on the contexts below. If the\n",
        "question cannot be answered using the information provided answer\n",
        "with \"I don't know\".\n",
        "\n",
        "###\n",
        "\n",
        "Contexts:\n",
        "{'##'.join(contexts)}\n",
        "\n",
        "###\n",
        "\n",
        "Question: Give me two examples of how to use OpenAI's GPT-3 model\n",
        "using Python from start to finish\n",
        "\n",
        "Answer: \"\"\"\n",
        "\n",
        "encoder_name = 'p50k_base'\n",
        "tokenizer = tiktoken.get_encoding(encoder_name)\n",
        "\n",
        "len(tokenizer.encode(prompt))"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "v396Uo2Aba33"
      },
      "source": [
        "When feeding this prompt into `text-davinci-003` it will use `412` of our maximum context window of `4097`, leaving us with `4097 - 412 == 3685` tokens for our completion.\n",
        "\n",
        "---\n",
        "\n",
        "*Not all OpenAI models use the `p50k_base` encoder, a table of different encoders for different models can be found [here](), as of this writing they are:*\n",
        "\n",
        "| Encoding name | OpenAI models |\n",
        "| --- | --- |\n",
        "| `gpt2` (or `r50k_base`) | Most GPT-3 models (and GPT-2) |\n",
        "| `p50k_base` | Code models, `text-davinci-002`, `text-davinci-003` |\n",
        "| `cl100k_base` | `text-embedding-ada-002` |\n",
        "\n",
        "---\n",
        "\n",
        "By default the maximum number of tokens used for completion is `256`. We can increase this upto the maximum calculated above of `3685`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 16,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8b0MPOw_ba33",
        "outputId": "2b9082a4-c38e-48cc-a813-a2315ad85e92"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "1. Import the `openai` library with pip, set the API key, and use the `Completion.create()` method to generate a response to a prompt: \n",
            "```import openai\n",
            "openai.api_key = 'YOUR_API_KEY'\n",
            "prompt = '<YOUR PROMPT>'\n",
            "res = openai.Completion.create(engine='text-davinci-003', prompt=prompt, max_tokens=100)\n",
            "print(res)```\n",
            "\n",
            "2. Install the LangChain library with `pip install langchain openai`, import the library, and initialize the model with the API key: \n",
            "```from langchain.llms import OpenAI\n",
            "openai = OpenAI(model_name='text-davinci-003', openai_api_key='YOUR_API_KEY')\n",
            "prompt = 'YOUR_PROMPT'\n",
            "print(openai(prompt))```\n"
          ]
        }
      ],
      "source": [
        "res = openai.Completion.create(\n",
        "    engine='text-davinci-003',\n",
        "    prompt=prompt,\n",
        "    temperature=0.0,\n",
        "    max_tokens=3685\n",
        ")\n",
        "\n",
        "print(res['choices'][0]['text'].strip())"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "RSVA6J_Yba33"
      },
      "source": [
        "The model doesn't need the full size of completion and doesn't try to fill the full space, but because we increased the value of `openai.max_tokens`, inference does take notably longer.\n",
        "\n",
        "If we exceed the maximum context window allowed, we'll see an error."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 17,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XqVb2qlqba33",
        "outputId": "0cb37f85-9346-4878-b021-5851e67743e9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "This model's maximum context length is 4097 tokens, however you requested 4098 tokens (412 in your prompt; 3686 for the completion). Please reduce your prompt; or completion length.\n"
          ]
        }
      ],
      "source": [
        "try:\n",
        "    res = openai.Completion.create(\n",
        "        engine='text-davinci-003',\n",
        "        prompt=prompt,\n",
        "        temperature=0.0,\n",
        "        max_tokens=3686\n",
        "    )\n",
        "except openai.InvalidRequestError as e:\n",
        "    print(e)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "UfY1mT8wba33"
      },
      "source": [
        "So it can be a good idea to integrate this type of check into our code if we expect to exceed the maximum context window at any point."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "ml",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.12"
    },
    "orig_nbformat": 4,
    "vscode": {
      "interpreter": {
        "hash": "b8e7999f96e1b425e2d542f21b571f5a4be3e97158b0b46ea1b2500df63956ce"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
